Guide: migrating from OSS to Cloud using BACKUP/RESTORE commands #4663
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@Blargian is attempting to deploy a commit to the ClickHouse Team on Vercel. A member of the Team first needs to authorize it. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dhtclk
requested changes
Nov 17, 2025
|
|
||
| ### Create an AWS IAM user {#create-an-aws-iam-user} | ||
|
|
||
| In this procedure, we'll be creating a service account user, not a login user. |
Collaborator
There was a problem hiding this comment.
nit - procedure wording sounds awkward here.
| 1. Log into the AWS IAM Management Console. | ||
|
|
||
| 2. In "users", select **Add users** | ||
| 2. In the `Users`, select `Create user` |
Collaborator
There was a problem hiding this comment.
Suggested change
| 2. In the `Users`, select `Create user` | |
| 2. In `Users`, select `Create user` |
or
Suggested change
| 2. In the `Users`, select `Create user` | |
| 2. In the `Users` menu, select `Create user` |
|
|
||
| ## Introduction {#introduction} | ||
|
|
||
| Before diving into the setup for secure S3 access, it is important to understand how this works. Below is an overview of how ClickHouse services can access private S3 buckets by assuming into a role within customers' AWS account. |
Collaborator
There was a problem hiding this comment.
Suggested change
| Before diving into the setup for secure S3 access, it is important to understand how this works. Below is an overview of how ClickHouse services can access private S3 buckets by assuming into a role within customers' AWS account. | |
| Before diving into the setup for secure S3 access, it is important to understand how this works. Below is an overview of how ClickHouse services can access private S3 buckets by assuming a role within customers' AWS account. |
| <Image img={s3_info} size="lg" alt="Obtaining ClickHouse service IAM Role ARN" border /> | ||
|
|
||
| ### Setting up IAM assume role {#setting-up-iam-assume-role} | ||
| ## Set up IAM assume role {#setting-up-iam-assume-role} |
Collaborator
There was a problem hiding this comment.
Suggested change
| ## Set up IAM assume role {#setting-up-iam-assume-role} | |
| ## Set up an IAM assume role {#setting-up-iam-assume-role} |
Comment on lines
+54
to
+59
| 4. Enter your bucket name in the input titled "Bucket Names". If your bucket URL is `https://ch-docs-s3-bucket.s3.eu-central-1.amazonaws.com/clickhouseS3/` then the bucket name is `ch-docs-s3-bucket`. | ||
|
|
||
| :::note | ||
| Do not put the full bucket ARN but instead just the bucket name only. | ||
| ::: | ||
|
|
Collaborator
There was a problem hiding this comment.
Suggested change
| 4. Enter your bucket name in the input titled "Bucket Names". If your bucket URL is `https://ch-docs-s3-bucket.s3.eu-central-1.amazonaws.com/clickhouseS3/` then the bucket name is `ch-docs-s3-bucket`. | |
| :::note | |
| Do not put the full bucket ARN but instead just the bucket name only. | |
| ::: | |
| 4. Enter your bucket name in the **Bucket Names** field. | |
| :::note[Use bucket name only] | |
| Enter the bucket name only, not the full bucket ARN or URL. For example, if your bucket URL is `https://ch-docs-s3-bucket.s3.eu-central-1.amazonaws.com/clickhouseS3/`, the bucket name is `ch-docs-s3-bucket`. | |
| ::: | |
Comment on lines
+193
to
+197
| Next you will need to create an access role. These steps are detailed in the guide ["Accessing S3 data securely"](/cloud/data-sources/secure-s3). | ||
| Follow the steps in that guide to obtain an access role ARN. | ||
|
|
||
| In ["How to create an S3 bucket and IAM role"](/integrations/s3/creating-iam-user-and-s3-bucket) you created | ||
| a policy for your S3 bucket. You'll now need to add the ARN you obtained in in ["Accessing S3 data securely"](/cloud/data-sources/secure-s3) from the output of the created stack to your bucket policy. |
Collaborator
There was a problem hiding this comment.
Suggested change
| Next you will need to create an access role. These steps are detailed in the guide ["Accessing S3 data securely"](/cloud/data-sources/secure-s3). | |
| Follow the steps in that guide to obtain an access role ARN. | |
| In ["How to create an S3 bucket and IAM role"](/integrations/s3/creating-iam-user-and-s3-bucket) you created | |
| a policy for your S3 bucket. You'll now need to add the ARN you obtained in in ["Accessing S3 data securely"](/cloud/data-sources/secure-s3) from the output of the created stack to your bucket policy. | |
| ### Set up S3 access | |
| To restore your backup from S3, you'll need to configure secure access between ClickHouse Cloud and your S3 bucket. | |
| 1. Follow the steps in ["Accessing S3 data securely"](/cloud/data-sources/secure-s3) to create an access role and obtain the role ARN. | |
| 2. Update the S3 bucket policy you created in ["How to create an S3 bucket and IAM role"](/integrations/s3/creating-iam-user-and-s3-bucket) by adding the role ARN from the previous step. | |
| Your updated bucket policy should look something like this: |
Comment on lines
+227
to
+229
| Specifying both the user ARN and the ClickHouse Cloud access user role ensures | ||
| that you will be able to both backup to the S3 bucket and later restore from it | ||
| using the Cloud access role. |
Collaborator
There was a problem hiding this comment.
Suggested change
| Specifying both the user ARN and the ClickHouse Cloud access user role ensures | |
| that you will be able to both backup to the S3 bucket and later restore from it | |
| using the Cloud access role. | |
| The policy includes both ARNs: | |
| - **IAM user** (`docs-s3-user`): Allows your self-managed ClickHouse cluster to back up to S3 | |
| - **ClickHouse Cloud role** (`ClickHouseAccess-001`): Allows your Cloud service to restore from S3 |
| You will be restoring your data into a new Cloud service. | ||
| Follow the steps below to create a new Cloud service. | ||
|
|
||
| 1. Go to [https://console.clickhouse.cloud/](https://console.clickhouse.cloud/) |
Collaborator
There was a problem hiding this comment.
For these sections with numbered steps let's use our vertical stepper when we're writing new guides or updating old ones.
|
|
||
| <Image img={backup_s3_bucket} size="md" alt="backup, data and metadata"/> | ||
|
|
||
| If you are performing a full migration then you can run the following command to backup the entire server: |
Collaborator
There was a problem hiding this comment.
Suggested change
| If you are performing a full migration then you can run the following command to backup the entire server: | |
| If you're performing a full migration then you can run the following command to backup the entire server: |
| - Quotas | ||
| - User-defined functions | ||
|
|
||
| If you are using a different CSP, you can use the `TO S3()` (for both AWS and GCP) and `TO AzureBlobStorage()` syntax. |
Collaborator
There was a problem hiding this comment.
Suggested change
| If you are using a different CSP, you can use the `TO S3()` (for both AWS and GCP) and `TO AzureBlobStorage()` syntax. | |
| If you're using a different CSP, you can use the `TO S3()` (for both AWS and GCP) and `TO AzureBlobStorage()` syntax. |
Collaborator
|
Hey Shaun - I've reviewed this and think we need to do some rewording and restructuring to align with docs standards for new or updated guides. A few things I'm thinking:
Happy to discuss or pair on this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds a guide on how to migrate from OSS to Cloud using BACKUP/RESTORE commands.
Also splits out the guide on how to set up an S3 bucket and IAM role into it's own page instead of a snippet and updates the images. The previous one is quite outdated.
Backup and restore docs are improved here: #4049
Checklist